This appendix provides details on describing data so that GrADS can read it. GrADS supports two basic data types: 1) gridded ; and 2) station or point observations. The data and meta data (or information about the data) are kept in separate files. The file which you open in GrADS is the data descriptor file (the meta data) or ".ctl" file. The .ctl is constructed to describe various data types and structures (e.g., binary and GRIB).
There is a new record in the data descriptor file:
options <keywords>
This obsolotes the "format" record and allows for a greater variety of binary data to be described and access (opened) by GrADS.
Some keywords:
options <yrev> <zrev> <sequential> <byteswapped> <template>
<big_endian> <little_endian>
sequential
specifies that the file was written in sequential unformatted I/O, where each record is an X/Y varying grid. Note that if you have only one X and one Y dimension in your file, each record in the file will be one element long (it may not be a good idea to write the file this way).
yrev
specifies that the Y, or latitude, dimension has been written in the reverse order from what GrADS has in the past assumed. An important thing to remember is that GrADS still presents the view that the data goes from south to north. The YDEF statement does not change; it still describes the transformation from a grid space going from south to north. The reversal of the Y axis is done as the data is being read from the data file.
zrev
indicates the data has been written into the file from top to bottom, rather than from bottom to top as GrADS has in the past assumed. The same considerations as Yrev apply.
The best way to insure independence of hardward for gridded data files is to specifiy the source platform of the data. This allows the data to worked on both type of hardware without having to worry about byte ordering.
big_endian
32-bit IEEE floats created on a big_endian platform (e.g., cray, sun, sgi and hp).
little_endian
32-bit IEEE floats created on a little_endian platform (e.g., iX86, and dec)
You may tell GrADS that your data file has a header. To do this, include the record in your descriptor file:
fileheader bytenum
where bytenum is the number of bytes in the header. GrADS will skip past this header, then treat the file as though it were a normal GrADS file after that point. This option is valid only for GrADS gridded data sets.
Up until now, you have been able to specify that the byte ordering of your data is the reverse of the standard for your workstation. This was done via the 'byteswapped' keyword. You may now indicate the actual byte ordering of the data:
options big_endian
or
options little_endian
If the data is already in the correct order, no conversion is performed. If you then move the data set to another machine, conversion will be automatically performed. These options were added to facilatate moving data files and descriptor files between machines.
You may also now specify:
options byteswapped
The byteswapped record is obsolote. The byteswapped option should be specified on the options record.
GrADS now allows you to handle many actual data files as one GrADS file, if the individual data files are in a GrADS readable format, and if the files are split along time. In the initial implementation, the time(s) that are in each file are indicated by the file name.
An example of this might be hourly data, where each 24 hours has been placed in a separate file. Each file is named this way:
1may92.dat
2may92.dat
etc.
You indicate to GrADS that there are multiple files in this time series by giving a substitution template as the file name:
dset %d1%mc%y2.dat
and giving an options record that looks like:
options template
and specifying the time range and increment in the tdef record:
tdef 72 linear 0z1may1993 1hr
GrADS will figure out automatically that there are 24 times in each file, and what file names coorospond to what times. As you display data, GrADS will only open one file at a time. As you change times such that another file is referred to, the open file is closed, and the new file is opened.
Valid subsitutions are:
%y2 - 2 digit year (last 2 digits)
%y4 - 4 digit year
%m1 - 1 or 2 digit month
%m2 - 2 digit month (leading zero if needed)
%mc - 3 character month abbreviation
%d1 - 1 or 2 digit day
%d2 - 2 digit day
%h1 - 1 or 2 digit hour
%h2 - 2 digit hour
This support works on all supported GrADS data types (GrADS gridded, GRIB, GrADS station data). If you specify file format options, the options must apply equally to each file.
The real-time data on the DECstations makes use of this new feature. See the data descriptor files:
/data/wx/grads/sa.ctl
/data/wx/grads/sareps.ctl
/data/wx/grads/wx.ctl
for additional examples.
The GrADS gridded data set is a direct access binary data set. It may contain any number of variables at specified longitude, latitude, vertical, and time intervals.
GrADS views this data set as a giant array -- with X (longitude) varying the fastest, then Y (latitude), then Z (vertical level), then the variable type, then T (time).
It is easier for us to think of the data set in terms of a sequence of horizontal grids, where longitude and latitude vary. Each horizontal grid represents a particular variable at a particular height and time. Each horizontal grid is the same size in any particular GrADS data set (if you have grids of different sizes, you must create seperate data sets).
These grids are written to the data set in the following order: starting with a particular variable, grids for each vertical level (at a particular time) are written out in ascending order. Then the grids for the next variable are written out. When all the grids at a particular time have been written, grids for the next time are written.
The format of this data set is thus exactly the same as the COLA Pressure History format, except: there are no date/time records, and latitude varies from south to north (not north to south as in the pressure history data).
Each binary gridded data set is described by a data descriptor file, essentially a table of contents for the binary data set. Following is an example of such a file:
DSET ua.dat
TITLE Upper Air Data
UNDEF -9.99E33
XDEF 80 LINEAR -140.0 1.0
YDEF 50 LINEAR 20.0 1.0
ZDEF 10 LEVELS 1000 850 700 500 400 300 250 200 150 100
TDEF 4 LINEAR 0Z10apr1991 12hr
VARS 5
slp 0 0 sea level pressure
z 10 0 heights
t 10 0 temps
td 6 0 dewpoints
u 10 0 u winds
v 10 0 v winds
ENDVARS
The data descriptor file is 'free format', ie each entry is blank delimited and may appear in any column. Comment records start with an asterisk ('*') in column 1. Comments may not appear in the list of variable records (between the vars and endvars records). Records may not be more than 80 characters long.
In this example, the binary data set is named ua.dat, the undefined, or missing, data value is -9.99e33, there are 80 grid points in the X direction, 50 in the Y direction, 10 levels, 4 times, and 5 variables. The variables z, t, u, and v have 10 levels, the variable td has 6 levels, and the variable slp has one level (see below for a more specific description of each entry).
Think in terms of the X and Y data points at one level for one variable at one time being a horizontal grid. This grid is exactly in the same storage order as a FORTRAN array, in this case an array DIMENSION A(80,50). The first dimension always varies from west to east, the second from south to north.
In the above example the horizontal grids would be written in the following order:
Time 1, Level ?, Variable slp
Time 1, Level 1000, Variable z
Time 1, Level 850, Variable z
then levels 700, 500, 400, 300, 250, 200, then
Time 1, Level 150, Variable z
Time 1, Level 100, Variable z
Time 1, Level 1000, Variable t
Time 1, Level 850, Variable t
then levels 700, 500, 400, 300, 250, 200, then
Time 1, Level 150, Variable t
Time 1, Level 100, Variable t
Time 1, Level 1000, Variable td
Time 1, Level 850, Variable td
Time 1, Level 700, Variable td
Time 1, Level 500, Variable td
Time 1, Level 400, Variable td
Time 1, Level 300, Variable td
Time 1, Level 1000, Variable u
Time 1, Level 850, Variable u
then levels 700, 500, 400, 300, 250, 200, then
Time 1, Level 150, Variable u
Time 1, Level 100, Variable u
Time 1, Level 1000, Variable v
Time 1, Level 850, Variable v
then levels 700, 500, 400, 300, 250, 200, then
Time 1, Level 150, Variable v
Time 1, Level 100, Variable v
Time 2, Level ?, Variable slp
Time 2, Level 1000, Variable z
Time 2, Level 850, Variable z
Time 2, Level 700, Variable z
Time 2, Level 500, Variable z
Time 2, Level 400, Variable z
.
.
.
etc
A description of each record in the GrADS data descriptor file follows:
DSET data-set-name
This entry specifies the name of the binary data set. It may be entered in mixed case.
If the binary data set is in the same directory as the data descriptor file, you may enter the filename in the data descriptor file without a full path name by prefixing it with a ^ character. For example, if the data descriptor file is:
/data/wx/grads/sa.ctl
and the binary data file is:
/data/wx/grads/sa.dat
you could use the following file name in the data descriptor file:
DSET ^sa.dat
instead of:
DSET /data/wx/grads/sa.dat
As long as you keep the two files together, you may move them to any directory without changing the entries in the data descriptor file.
TITLE string
A brief description of the contents of the data set. This will be displayed during a QUERY command, so it is helpful to put meaningful information here.
UNDEF value
The undefined, or missing, data value. GrADS operations and graphics routines will ignore data with this value from this data set.
OPTIONS BYTESWAPPED
Indicates the binary data file is in reverse byte order from the normal byte order of the machine. This would happen if you sent a file in binary fromat from, for example, a Sun to a PC. Putting this keyword in the descriptor file tells GrADS to swap the byte order as the data is being read.
XDEF number <LINEAR start increment>
<LEVELS value-list>
Defines the mapping between grid values and longitude. Specifically:
number -- the number of grid values in the X direction, specified as an integer number. Must be >= 1.
LINEAR or LEVELS -- Indicates the grid mapping type.
For LINEAR:
start -- the starting longitude, or the longitude for X = 1. Specified as a floating point value, where negative indicates degrees west.
increment -- the spacing between grid value in the X direction. It is assumed that the X dimension values go from west to east. Specified as a positive floating value.
For LEVELS:
value-list -- List of 'number' values representing the longitude of each X dimension. May start and continue on the next record in the descriptor file (records may not be > 80 characters). There must be at least 2 levels (otherwise use LINEAR mapping).
YDEF number mapping start <increment>
<LEVELS value-list>
Defines the mapping between grid values and latitude. Specifically:
number -- the number of grid values in the X direction, specified as an integer number.
mapping -- mapping type, specified as a keyword.
Valid are:
LINEAR -- Linear mapping
GAUSR15 -- Gaussian R15 latitudes
GAUSR20 -- Gaussian R20 latitudes
GAUSR30 -- Gaussian R30 latitudes
GAUSR40 -- Gaussian R40 latitudes
start -- For LINEAR mapping, the starting latitude, ie the latitude for Y = 1, and is specified as a floating point value, with negative indicating degrees south. For GAUSRxx mapping, the start value indicates the first gaussian grid number, where 1 would be the southernmost gaussian grid latitude.
increment -- the spacing between grid values in the Y direction. It is assumed that the Y dimension values go from south to north. Specified as a positive floating point value. Used only for LINEAR mapping.
For LEVELS:
value-list -- List of 'number' values representing the latitude of each X dimension. May start and continue on the next record in the descriptor file (records may not be > 80 characters). There must be at least 2 levels (otherwise use LINEAR mapping).
Examples of specifying GAUSRxx mapping:
YDEF 20 GAUSR40 15
Indicates that there are 20 Y dimension values which start at Gaussian Latitude 15 (64.10 south) on the Gaussian R40 grid. Thus the 20 values would coorospond to Latitudes:
-64.10, -62.34, -60.58, -58.83, -57.07, -55.32, -53.56,
-51.80, -50.05, -48.29, -46.54, -44.78, -43.02, -41.27,
-39.51, -37.76, -36.00, -34.24, -32.49, -30.73
YDEF 102 GAUSR40 1
The entire gaussian grid is present, starting at the southernmost latitude (-88.66).
ZDEF number mapping <start increment>
<value-list>
Defines the mapping between grid values and pressure level. Specifically:
number -- the number of grid values in the X direction, specified as an integer number.
mapping -- mapping type, specified as a keyword.
Valid are:
LINEAR -- Linear mapping
LEVELS -- Arbitrary pressure levels
start -- when mapping is LINEAR, this is the starting value, or the value when Z=1.
increment -- when mappeing is LINEAR, the increment in the Z direction, or from lower to higher. This may be a negative value, for example:
ZDEF 10 LINEAR 1000 -100
indicating that the data is for levels 1000, 900, 800, 700, etc.
value-list -- when the mapping is LEVELS, the specific levels are simply listed in ascending order. If there is only one level, use LINEAR, since LEVELS implies at least two levels.
TDEF number LINEAR start-time increment
Defines the mapping between grid values and time. Specifically:
number -- the number of times in the data set. Specified as an integer number.
start-time -- The starting date/time value, specified in GrADS absolute date/time format. This is the value when T=1. The date/time format is:
hh:mmZddmmmyyyy
where:
hh = hour (two digit integer)
mm = minutes (two digit integer)
dd = day (one or two digit integer)
mmm = month (jan, feb, mar, apr, may, jun, jul, aug, sep, oct, nov, dec)
yyyy = year (two or four digit integer. two digits implies a year between 1950 and 2049).
If not specified, hh defaults to 00, mm defaults to 00, and dd defaults to 1. The month and year must be specified. No intevening blanks are allowed in a GrADS absolute date/time.
Examples:
12Z1JAN1990
14:20Z22JAN1987
JUN1960
increment -- time increment. Specified in GrADS time increment format:
vvkk
where:
vv = an integer number, 1 or 2 digits
kk = an increment keyword,
mn = minutes
hr = hours
dy = days
mo = months
yr = year
Examples:
20mn -- increment is 20 minutes
1mo -- increment is 1 month
2dy -- increment is 2 days
Some examples of a TDEF statement:
TDEF 24 LINEAR 00Z01JUN1987 1HR
The data set has 24 times, starting at 00Z on 1 Jun, 1987, with an increment of 1 hour.
TDEF 30 LINEAR 2JUN1988 1DY
The data set has 30 times, starting at 00Z on 2 Jun, 1988, with an increment of 1 day.
VARS number
Indicates the start of the records describing the variables in the data set.
number -- the number of variable records
Each variable record is in the following format:
abrev levs units description
abrev -- a 1 to 8 character abbreviation for this variable. This abreviation must start with an alphabetic character (a-z) and be composed of alphabetic characters and numbers. This abbreviation will be the "name" the variable is accessed by from within GrADS.
levs -- an integer value specifying the number of levels this variable has in the data set. It may not exceed the number of levels in the ZDEF statement. A levs value of 0 indicates this variable has one "level" that does not coorospond to a vertical level. An example would be a surface variable.
units - Reserved for future use. Put a value of 99 here.
description - A text description of the variable, max 40 characters.
After the last variable record comes the ENDVARS statement. This ends the GrADS data descriptor file.
Station data sets are written to a binary file one report at a time. The only ordering required is that the station reports be grouped within the file into some time interval. For example, the time interval for upper air observations might be 12 hours.
Variables within each report are split into two groupings. Each variable is either a surface variable, thus can be reported at most once per report, or it is a level dependent variable, thus can be reported at a number of different levels within one report.
Byte-ordering control for station data files: You may now specify byteswapping (byteswapped, big_endian, or little_endian) for station data files. The stnmap utility, and GrADs, will perform the necessary conversion. Station map files must still be created on the machine where they are to be used.
Station map files (and GRIB index files) will be made portable before Version 1.4 is released in production.
The format of a station report in the binary station data file is:
- A header which provides information about the location of the station.
- Surface variables, if any
- Level dependent variables, if any
The header is described by the following C language data structure:
struct rpthdr {
char id[8]; /* Character station id */
float lat; /* Latitude of report */
float lon; /* Longitude of report */
float t; /* Time in relative grid units */
int nlev; /* Number of levels following */
int flag; /* Level independent var set flag */
};
A detailed description of each header entry follows:
id - The station identifier. This is a 1 to 7 character identifier that should identify the station uniquely. It may be assigned arbitrarily; ie. the stations could be numbered in some arbitrary order.
lat - The Y dimension location of the station in world coordinates, typically latitude.
lon - The X dimension location of the station in world coordinates, typically longitude.
t - The time of this report, in relative grid units. This refers to the way the stations are grouped in time. For example, if you are working with surface airways reports, you would probably have a time grouping interval of one hour. If you wanted to treat the report times of each report as being exactly on the hour, you would set t to 0.0. If the report was for 12:15pm, and you were writing the time group for 12pm, you would set t to be 0.25. Thus, t would typically have the range of -0.5 to 0.5.
nlev - Number of data groups following the header. This is the count of the one surface group, if present, plus the number of level dependent groups. Is set to zero to mark the end of a time group in the file.
flag - If zero, there are no surface variables following the header. If one, then there are surface variables following the header.
Following the header, the data for this report is written. The first group of data would be all the surface variables if present. Whether or not the surface variable (if any) are present is determined by the flag in the header. If present, then all the surface varialbes must be written -- missing variables should have the missing data value provided. Thus, each surface variable group will be the same size for each report in the file.
The surface variables are written out as floating point numbers. The ordering of the variables must be the same in each report, and is the ordering that will be given in the data descriptor file.
Following the surface variable group, any number of level dependent groups may be written. The number of total data groups is provided in the header. Each level dependent group must have all the level dependent variables present, even if they are filled with the missing data value. Thus, each level dependent group will be the same size for all levels and all reports in the file.
The level dependent group is written out as follows:
level -- floating point value giving the Z dimension value in world coordinates for this level.
variables -- The level dependent variables for this level.
After all the reports for one time grouping have been written, a special header (with no data groups) is written to indicate the end of the time group. The header has an nlev value of zero. The next time group may then start immediately after. A time group with no reports would still contain the time group terminator header record (ie, two terminators in a row).
GrADS station data files must be written as UNIX stream data sets without any imbedded record descriptor information. This is easily done from a C program. From a FORTRAN program, it usually requires a system-dependent option in the OPEN statement. For example, in DEC FORTRAN one can use the
RECORDTYPE='STREAM'
option to avoid having record descriptor information imbedded in the output file. Examples of C and FORTRAN programs to create station data sets are provided later in this document.
The format for the data descriptor file for station data is similar to the format for a gridded data set. An example of a station data descriptor file is:
dset ^ua.reps
dtype station
stnmap ^ua.map
undef -999.0
title Real Time Upper air obs
tdef 10 linear 12z18jan1992 12hr
vars 12
slp 0 99 SLP
ts 0 99 Temps
ds 0 99 Dewpoints
us 0 99 U Winds
vs 0 99 V Winds
z 1 99 Heights
t 1 99 Temps
d 1 99 Dewpoints
u 1 99 U Winds
v 1 99 V Winds
endvars
Note the differences between this descriptor file and a grid descriptor file:
DTYPE record -- specify a data type of: station.
STNMAP record -- gives the file name of the station mapping file. This file is created by the
stnmap utility, which will be described later.
XDEF, YDEF, ZDEF records -- not specified.
TDEF record -- describes the time grouping interval and the number of time groups in the file.
VAR records -- surface variables must come first, and are given a zero for the number-of-levels field. Level dependent variables are listed after the surface variables, and are given a one in the number-of-levels field.
Once the data set has been written, and the descriptor file created, you should then create the station map file by running the stnmap utility. This utility writes out hash table and/or link list information that allows GrADS to access the report data more efficiently. The utility will prompt for the name of the data descriptor file.
If you change the data file -- perhaps by appending another time group -- you will also have to change the descriptor file to reflect the changes -- the new number of times for example -- and then rerun the stnmap utility.
On a workstation, the binary GrADS data sets need to be created as a 'stream' data set, ie, it should not have the normal FORTRAN record descriptor words imbedded in it. This can be done from FORTRAN using direct access I/O:
REAL Z(72,46,16)
.
.
OPEN (8,FILE='grads.dat',FORM='UNFORMATTED',
& ACCESS='DIRECT',RECL=72*46)
.
.
IREC=1
DO 10 I=1,18
WRITE (8,REC=IREC) ((Z(J,K,I),J=1,72),K=1,46)
IREC=IREC+1
10 CONTINUE
This example writes out 16 levels of one variable to a file in direct access format. We are really writing the data out sequentially, and using direct access to avoid having the record descriptor words written. There may be options in your compiler to do this more directly, or you ay wish to write the data using a C program.
Another simple sample might be:
REAL X(100)
DO 10 I=1,100
X(I)=I
10 CONTINUE
OPEN (8,FILE='samp.dat',FORM='UNFORMATTED',ACCESS='DIRECT',
& RECL=100)
WRITE (8,REC=1) X
STOP
END
The associated descriptor file:
DSET samp.dat
TITLE Sample Data Set
UNDEF -9.99E33
XDEF 100 LINEAR 1 1
YDEF 1 LINEAR 1 1
ZDEF 1 LINEAR 1 1
TDEF 1 LINEAR 1JAN2000 1DY
VARS 1
x 0 99 100 Data Points
ENDVARS
Once created, you can use this data set to experiment with GrADS data functions, such as:
display sin(x/50)
Lets say you have a data set with monthly rainfall:
Year Month Stid Lat Lon Rainfall
1980 1 QQQ 34.3 -85.5 123.3
1980 1 RRR 44.2 -84.5 87.1
1980 1 SSS 22.4 -83.5 412.8
1980 1 TTT 33.4 -82.5 23.3
1980 2 QQQ 34.3 -85.5 145.1
1980 2 RRR 44.2 -84.5 871.4
1980 2 SSS 22.4 -83.5 223.1
1980 2 TTT 33.4 -82.5 45.5
.
.
A FORTRAN program in DEC FORTRAN to write this data set in GrADS format might be:
CHARACTER*8 STID
C
OPEN (8,NAME='rain.ch')
OPEN (10,NAME='rain.dat',FORM='UNFORMATTED',
& RECORDTYPE='STREAM')
C
IFLAG = 0
C
C Read and Write
C
10 READ (8,9000,END=90) IYEAR,IMONTH,STID,RLAT,RLON,RVAL
9000 FORMAT (I4,3X,I2,2X,A8,3F8.1)
IF (IFLAG.EQ.0) THEN
IFLAG = 1
IYROLD = IYEAR
IMNOLD = IMONTH
ENDIF
C
C If new time group, write time group terminator.
C Assuming no empty time groups.
C
IF (IYROLD.NE.IYEAR.OR.IMNOLD.NE.IMONTH) THEN
NLEV = 0
WRITE (10) STID,RLAT,RLON,TIM,NLEV,NFLAG
ENDIF
IYROLD = IYEAR
IMNOLD = IMONTH
C
C Write this report
C
TIM = 0.0
NLEV = 1
NFLAG = 1
WRITE (10) STID,RLAT,RLON,TIM,NLEV,NFLAG
WRITE (10) RVAL
GO TO 10
C
C On end of file write last time group terminator.
C
90 CONTINUE
NLEV = 0
WRITE (10) STID,RLAT,RLON,TIM,NLEV,NFLAG
STOP
END
For a different compiler, you would need to determine the appropriate OPEN statement to write a stream data set.
An equivalent C program might be:
#include <stdio.h>
/* Structure that describes a report header in a stn file */
struct rpthdr {
char id[8]; /* Character station id */
float lat; /* Latitude of report */
float lon; /* Longitude of report */
float t; /* Time in relative grid units */
int nlev; /* Number of levels following */
int flag; /* Level independent var set flag */
} hdr;
main () {
FILE *ifile, *ofile;
char rec[80];
int flag,year,month,yrsav,mnsav,i;
float val;
/* Open files */
ifile = fopen ("rain.ch","r");
ofile = fopen ("rain.dat","wb");
if (ifile==NULL || ofile==NULL) {
printf ("Error opening files\n");
return;
}
/* Read, write loop */
flag = 1;
while (fgets(rec,79,ifile)!=NULL) {
/* Format conversion */
sscanf (rec,"%i %i ",&year,&month);
sscanf (rec+20," %g %g %g",&hdr.lat,&hdr.lon,&val);
for (i=0; i<8; i++) hdr.id[i] = rec[i+11];
/* Time group terminator if needed */
if (flag) {
yrsav = year;
mnsav = month;
flag = 0;
}
if (yrsav!=year || mnsav!=month) {
hdr.nlev = 0;
fwrite (&hdr,sizeof(struct rpthdr), 1, ofile);
}
yrsav = year;
mnsav = month;
/* Write this report */
hdr.nlev = 1;
hdr.flag = 1;
hdr.t = 0.0;
fwrite (&hdr,sizeof(struct rpthdr), 1, ofile);
fwrite (&val,sizeof(float), 1, ofile);
}
hdr.nlev = 0;
fwrite (&hdr,sizeof(struct rpthdr), 1, ofile);
}
Once the binary data file has been written, create the descriptor file. It would look something like this:
dset rain.dat
dtype station
stnmap rain.map
undef -999.0
title Rainfall
tdef 12 linear jan1980 1mo
vars 1
p 0 99 Rainfall
endvars
Then run the stnmap utility to create the station map file. You can then open and display this data from within GrADS.